{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "81942eb1",
   "metadata": {},
   "source": [
    "# Course 2 Project: ML Pipeline for Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f241775",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "In this project, you'll use data related to microeconomic indicators and historical stock prices to explore the data engineering pipline. You'll get to practice:\n",
    "\n",
    "* Data ingestion \n",
    "* Data cleaning\n",
    "* Data imputation\n",
    "* Exploratory data analysis (EDA) through charts and graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f12b277",
   "metadata": {},
   "source": [
    "## Packages\n",
    "\n",
    "You'll use `pandas` and `matplotlib`, which were covered in the course material, to import, clean, and plot data. They have been installed in this workspace for you. If you're working locally and you installed Jupyter using Anaconda, these packages will already be installed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "882241d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9fcd352",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "The first step in a data engineering pipeline for machine learning is to ingest the data that will be used. For this project, data is hosted on a public GitHub repo. \n",
    "\n",
    "Your tasks:\n",
    "* Import data from the provided GitHub repo using `pandas`\n",
    "* Verify that the data has been imported correctly into `pandas` dataframes. Use methods like `head()` and `info()`\n",
    "* You may need to change column names to make them easier to work with\n",
    "* You may need to cast datetime data to the `datetime` format using `pandas` `to_datetime()` method\n",
    "\n",
    "Data files to import: \n",
    "1. GDP\n",
    "2. Inflation \n",
    "3. Apple stock prices \n",
    "4. Microsoft stock prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd119016",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load historical GDP data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf36d7c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the first few rows of data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a2478c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the historical stock price data for Apple and Microsoft\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9e9a77a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the first few rows of data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c9bed17",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use methods like .info() and .describe() to explore the data\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06c8d9dc",
   "metadata": {},
   "source": [
    "## Data preprocessing: Check for missing data and forward fill\n",
    "\n",
    "Check the Apple historical prices for missing data. Check for missing data in all columns. If there's data missing, use a forward fill to fill in those missing prices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4807173e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for nulls \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e4d6f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Forward fill any missing data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f82d3b2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check again for nulls after using forward fill\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2614c615",
   "metadata": {},
   "source": [
    "## Data preprocessing: Remove special characters and convert to numeric/datetime\n",
    "\n",
    "The next step in the data engineering process is to standardize and clean up data. In this step, you'll check for odd formatting and special characters that will make it difficult to work with data as numeric or datetime. \n",
    "\n",
    "In this step:\n",
    "* Create a function that takes in a dataframe and a list of columns and removes dollar signs ('$') from those columns\n",
    "* Convert any columns with date/time data into a `pandas` `datetime` format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a47537af",
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_dollar_columns_to_numeric(df, numeric_columns):\n",
    "    '''\n",
    "        Removes dollar signs ('$') from a list of columns in a given dataframe AND casts the columns to a numeric datatype.\n",
    "        Updates dataframe IN PLACE.\n",
    "        \n",
    "        Inputs:\n",
    "            df: dataframe to be operated on\n",
    "            numeric_columns: columns that should have numeric data but have dollar signs currently\n",
    "            \n",
    "        Returns:\n",
    "            None - changes to the dataframe can be made in place\n",
    "    '''\n",
    "    \n",
    "    \n",
    "    pass "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "055f8621",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use convert_dollar_columns_to_numeric() to remove the dollar sign from the Apple and Microsoft datasets\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efea17ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use pandas's to_datetime() to convert any columns that are in a datetime format\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2089e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use .info() and check the type of each column to ensure that the above steps worked\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95ad8e72",
   "metadata": {},
   "source": [
    "## Data preprocessing: Align datetime data\n",
    "\n",
    "Data engineering includes changing data with a datetime component if needed so that different time series can be more easily compared or plotted against each other. \n",
    "\n",
    "In this step:\n",
    "\n",
    "* Align the inflation date so that it falls on the last day of the month instead of the first\n",
    "\n",
    "Helpful hints: \n",
    "* Use the `pandas` `offsets` method using `MonthEnd(0)` to set the 'Date' column to month-end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe782db6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Align inflation data so that the date is the month end (e.g. Jan 31, Feb 28/28)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e072f4a1",
   "metadata": {},
   "source": [
    "## Data preprocessing: Upsample, downsample and interpolate data\n",
    "\n",
    "Inflation data is presented monthly in this dataset. However, for some models, you may need it at a quarterly frequency, and for some models you may need it at a quarterly frequency. \n",
    "\n",
    "In this step:\n",
    "* Create a new quarterly inflation dataframe by downsampling the monthly inflation data to quarterly using the mean (e.g. for quarter 1 in a given year, use the average values from January, February, and March)\n",
    "* Create a new weekly inflation dataframe by upsampling the monthly inflation data. For this, you'll need to use `resample` and then you'll need to `interpolate` to fill in the missing data at the weekly frequency\n",
    "\n",
    "Note that you may need to change the index for some of these operations! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28fea2f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upsample and interpolate from monthly to weekly \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "635647a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Downsample from monthly to quarterly \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ea82443",
   "metadata": {},
   "source": [
    "## Data preprocessing: Normalize/standardize a feature\n",
    "\n",
    "Economic time series data often involve variables measured on different scales (e.g., GDP in trillions of dollars, inflation in percentage points). Standardizing these variables (typically by subtracting the mean and dividing by the standard deviation) puts them on a common scale, allowing for meaningful comparisons and analyses.\n",
    "\n",
    "Your task:\n",
    "* Standardize the GDP data. You may do this manually by subtracting the mean and dividing by the standard deviation, or you may use a built-in method from a library like `sklearn`'s `StandardScaler`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f88d246",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standardize the GDP measure\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4801c53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the dataframe to make sure the calculation worked as expected \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f1a89d2",
   "metadata": {},
   "source": [
    "## EDA: Plotting a time series of adjusted open vs close price\n",
    "\n",
    "As part of your EDA, you'll frequently want to plot two time series on the same graph and using the same axis to compare their movements. \n",
    "\n",
    "Your task:\n",
    "* Plot the Apple open and close price time series on the same chart **for the last three months only**. Be sure to use a legend to label each line\n",
    "\n",
    "**NOTE**: This is a large dataset. If you try to plot the entire series, your graph will be hard to interpret and may take a long time to plot. Be sure to use only the most recent three months of data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b490f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get max date in timeseries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03b51a20",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the max date calculated above to get the last three months of data in the dataset \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "039404e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot time series of open v. close stock price for Apple using the last 3 months of data\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc0070c6",
   "metadata": {},
   "source": [
    "## EDA: Plotting a histogram of a stock's closing price in the last three months\n",
    "\n",
    "One way to see how much a stock's price generally moves is to plot the frequency of closing prices over a set time period. \n",
    "\n",
    "Your task:\n",
    "* Use the **last three months** of Apple stock data and plot a histogram of closing price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b48760fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the histogram of Apple's closing price over the last 3 months \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01e1b22c",
   "metadata": {},
   "source": [
    "## Calculating correlation between a stock price and a macroeconomic variable \n",
    "\n",
    "Inflation affects the purchasing power of money and can influence corporate profits, interest rates, and consumer behavior. By analyzing the correlation between stock prices and inflation, one can gauge how inflationary trends impact stock market performance. For instance, high inflation might erode profit margins and reduce stock prices, while moderate inflation might indicate a growing economy, benefiting stocks.\n",
    "\n",
    "Your task:\n",
    "* Plot a heatmap that shows the correlation between Microsoft and Apple returns and inflation \n",
    "\n",
    "This will require several steps:\n",
    "1. Calculate the returns for Apple and Microsoft and the change in monthly inflation (use the `pct_change` method for each)\n",
    "2. Interpolate the daily stock returns data to monthly so it can be compared to the monthly inflation data\n",
    "3. Merge the stock returns (Apple and Microsoft) and inflation data series into a single dataframe\n",
    "4. Calculate the correlation matrix between the Apple returns, Microsoft returns, and inflation change\n",
    "5. Plot the correlation matrix as a heatmap"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38842ed",
   "metadata": {},
   "source": [
    "### 1. Calculate returns for Microsoft / Apple and the monthly change in inflation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a112dc12",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate daily returns for Apple and Microsoft and the percent change in inflation from month to month\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8aa1b91",
   "metadata": {},
   "source": [
    "### 2. Interpolate stock returns from daily to monthly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9300a530",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "268272e6",
   "metadata": {},
   "source": [
    "### 3. Merge the dataframes and calculate / plot the correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0defb97b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e1352272",
   "metadata": {},
   "source": [
    "### 4. Calculate the correlation matrix between the Apple returns, Microsoft returns, and inflation change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8176f9ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate correlation matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b0ac07",
   "metadata": {},
   "source": [
    "### 5. Plot the correlation matrix as a heatmap "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f731a65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot heatmap\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "624a2410",
   "metadata": {},
   "source": [
    "## Calculating rolling volatility (standard deviation) of a stock's price for last 3 months\n",
    "\n",
    "Volatility is a measure of the dispersion of returns for a given security. By calculating rolling volatility, investors can assess the risk associated with a stock over time: Higher volatility indicates higher risk, as the stock's price is more likely to experience significant fluctuations. In portfolio optimization, understanding the volatility of individual stocks and how it changes over time is crucial for diversification and optimization. By analyzing rolling volatility, investors can adjust their portfolios to maintain a desired risk level, potentially improving the risk-return profile.\n",
    "\n",
    "One possible way to calculate volatility is by using the standard deviation of returns for a stock over time.\n",
    "\n",
    "Your task:\n",
    "* Calculate the weekly rolling standard deviation for Apple's closing price\n",
    "* Plot the calculated rolling weekly volatility of Apple's closing price against Apple's closing price. Plot these **on the same chart, but using different y-axes**\n",
    "\n",
    "Helpful hints:\n",
    "* You'll need to use the `pandas` `rolling()` method with a given `window_size` parameter to make it a _weekly_ rolling calculation\n",
    "* Use **only the last three months of data**; data much older than this may not be as useful for portfolio optimization\n",
    "* You'll need to create two axes on the matplotlib figure to be able to use two different y-axes (one for the closing price and one for the rolling volatility calculated here)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4da9eb94",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the window size for the rolling calculation (e.g., one week)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c5142b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate rolling one-week volatility\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92a9c0b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the calculated rolling weekly volatility of Apple's closing price against Apple's closing price\n",
    "# Plot these on the same chart, but using different y-axes\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2cf3c78",
   "metadata": {},
   "source": [
    "## Export data\n",
    "\n",
    "Now that you have preprocessed your data, you should save it in new csv files so that it can be used in downstream tasks without having to redo all the preprocessing steps. \n",
    "\n",
    "Your task:\n",
    "* Use `pandas` to export all modified datasets back to new CSV files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d67d29c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
